Corpus-Based Adaptation Mechanisms for Chinese Homophone Disambiguation

Based on the concepts of bzd~rectwnal converswn and automahc evaluatzon, we propose two user. adaptation mechanzsms, character-preference learn. in9 and pseudo-word learning, for resolving Chinese homophone ambiguities in syllable-to.character conversion. The 1991 Umted Daily corpus of approximately 10 million Chinese characters ts used for extraction of 10 reporter-specific article databases and .[or computat,on of word frequencies and character higrams. Ezpemments show that ~0.5 percent (testing sets) to 71.8 percent (trammg sets) of conversion er. rots can be eliminated through the proposed mechanisms. These concepts are thus very useful tn apphcattons such as Chinese znput methods and speech recognition systems. 1 I n t r o d u c t i o n Corpus-based Chinese NLP research has been very active in the recent years as more and more computer readable Chinese corpora are available. Reported corpus-based NLP applications [10] include machine translation, word segmentation, character recognition, text classification, lexicography, and spelling checker. In this paper, we will describe our work on adaptive Chinese homophone disambiguation (also known as phonetic-input-to-character conversion or phonetic decoding) using part of the 1991 United Daily (UD) corpus of approximately 10 million Chinese characters (Hanzi). It requires a coding method, structural or phonetic, to input Chinese characters into a computer, since there are more than I0,000 of them in common use. In the literature [3,7], there are several hundred different coding methods for this purpose. For most users, phonetic coding (Pinyin or Bopomofo) is the choice. To input a Chinese character, the user simply keys in its corresponding phonetic code. It is easy to learn, but suffers from the homophone problem, i.e., a phonetic code corresponding to several different characters. Therefore, the user needs to choose the desired character from a (usually long) list of candidate characters. It is inefficient and annoying. So, automatic homophone disambiguation is highly desirable. Several disambiguation approaches have been reported in the literature [3, 7]. Some of them have even been realized in commercial input methods, e.g., ttanin, WangXing, Going. However, the accuracies of these disambiguators are not satisfactory. In this paper, we propose a corpus-based adaptation method for improving the accuracy of homophone disambiguation. For homophone disambiguation, what we need as input is syllable (phonetic code) corpora instead of text corpora. For adaptation, what we need is personal corpora instead of general corpora (such as the UD corpus). Thus, we first design a selection procedure to extract articles by individual reporters. Ten personal corpora were set up in this way. An additional domain-specific corpus, translated AP news, was built up similarly. Then, we design a highly-reliable (99.7% correct) character-tosyllable converter [I] to transfer the text corpora into syllable corpora. Our baseline disambiguator is rather conventional, composed of a word-lattice searching module, a path scorer, and a lexicon-driven word hypothesizer. Using the original text corpora and the corresponding syllable corpora, we propose a user-adaptation method, applying the concept of bidirectional conversion [I] and automatic evaluation [2]. The adaptation method includes two parts: character-preference learning and pseudo word learning. Given a personal corpus (i.e., sample text), the adaptation pro-